Evaluation metrics for a translation memory system

This paper concerns the methods of evaluation of a computational translator’s aid known as a Translation Memory (TM). A typical TM system selects candidate translations into a target language by obtaining matches to an input source-language sentence from a database comprising source–target pairs of previously translated sentences. We describe a metric for evaluating TMs and illustrate it with a case study based on a particular commercial TM system and a database of French–English sentence pairs specifically created from a previously translated text. This metric builds on definitions of the familiar parameters of precision and recall, and involves a measure of the ‘usefulness’ to a translator of the selections made by the system. Several variations on these parameters and suggested weighting schemes are discussed in the search for a suitable metric, resulting in a ‘weighted precision’ measure based on the number of keystrokes needed to correct the proposed text. In combination with appropriate measures of recall, parameters which could serve both as numerical optimisation criteria to be employed by the translator in seeking the most efficient use of a TM tool, and as means of carrying out quantitative evaluations of such systems are defined. Copyright  1999 John Wiley & Sons, Ltd.